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A  Useful  Methodology  for  Cost-Benefit  Evaluations 
of  Cognitive  Process  Improvements 
in  Complex  Command  and  Control  (C2)  Endeavors 

Edmundo  O.  Acosta,  Ph.D.,  Wes  Nolden,  Ph.D.,  and  Tom  Gross,  M.A. 

ABSTRACT 

Proposed  solutions  to  identified  C2  gaps  can  be  at  a  disadvantage  in  the  fierce 
competition  for  scarce  dollars  because  they  are  often  considered  “soft”  benefits  (e.g. , 
augmenting  human  cognition  to  improve  performance).  This  paper  describes  a 
straightforward  method  to  quantify  and  express  “soft”  benefits  in  monetary  terms. 

Within  a  conventional  cost-benefit  structure,  the  proposed  approach  offers  a  tool  to 
demonstrate  “soft”  cognitive  improvements  in  “hard”  dollars. 

This  methodology  is  in  use  to  quantify  soft  benefits  of  the  Work-Centered  Interfaced 
Distributed  Environment  (WIDE),  a  proposed  enhancement  to  the  Global  Decision 
Support  System.  Evaluative  experiments  (specific  to  WIDE)  demonstrate  soft  cognitive 
improvements  in  measurable  values.  These  metrics  are  then  transformed  into  dollar 
values,  and  incorporated  into  the  benefits  portion  of  the  cost-benefit  structure.  In 
addition,  other  potential  soft  benefits  from  WIDE  are  quantified  and  monetized  through 
cost-estimating  techniques.  Cost-Benefit  Ratios  and  Retum-on-Investment  calculations 
are  easily  generated. 

This  cost-benefit  approach  is  a  tool  for  advocates  to  (1)  frame  discussions  of  C2 
solutions,  (2)  produce  quantified  responses  to  “what-if  ’  challenges,  and  (3)  to  argue  for 
the  funding  of  “soft”  proposals  in  “hard”  dollar  terms 

Introduction 

Capabilities  based  planning  (CBP)  is  the  process  of  looking  at  operations  and  acquisition  in 
light  of  desired  military  effects.  It  means  that  the  desired  military  effects  of  an  operation  or 
mission  are  considered  first  and  then  the  capabilities  required  to  achieve  those  effects  are 
reviewed  and  analyzed.  It  is  a  strategy  to  task,  top-down  approach  to  developing  required 
capabilities  to  support  the  achievement  of  desired  Joint  warfighting  effects. 

The  functional  analysis  to  achieve  the  CBP  goals  is  comprised  of  three  major  components. 
First  is  the  Functional  Area  Analysis  (FAA)  which  identified  operational  tasks,  conditions, 
and  standards  needed  to  accomplish  the  required  military  objectives  defined  by  needed 
capabilities^  1)  Once  these  capabilities  have  been  identified  and  set  down  in  a  capabilities 
list,  the  process  moves  to  the  Functional  Needs  Analysis  (FNA).(2)  The  FNA  assesses  the 
ability  of  current  and  programmed  capabilities  to  accomplish  the  tasks  and  identifies  any 
deficiencies.  Finally,  the  Functional  Solutions  Analysis  (FSA)  uses  an  operationally  based 
assessment  of  doctrine,  organization,  training,  materiel,  leadership  and  education, 
personnel,  and  facilities  (DOTMLPF)  approach  to  solve  the  identified  capability  gaps  or 
deficiencies.  It  is  within  the  FSA  that  a  cost  benefit  evaluation  is  accomplished.  As  part  of 
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the  CBP  process,  the  cost  benefit  evaluation  allows  decision  makers  to  evaluate  a  proposed 
solution  to  an  identified  capability  deficiency  with  an  eye  toward  achieving  capability  goals 
that  provide  the  greatest  return  on  investment. 

Within  the  C2  realm,  many  of  the  solutions  are  what  may  be  called  “soft”  solutions.  The 
reason  they  are  called  “soft”  is  because  often  the  improvements  are  in  time  efficiency, 
behavioral  improvements,  and  other  areas  of  work  that  are  less  easily  measured  than,  say, 
the  efficiency  of  a  new  aircraft  engine.  Combined  with  appropriate  investigative 
measurement,  cost  benefit  evaluations  can  provide  solid  rationale  for  C2  system  investment 
by  presenting  the  financial  case  for  “soft”  solutions.  This  paper  will  present  WIDE  and  the 
Global  Response  and  Synchronization  (GRS)  application  as  technologies  that  improve  C2 
operator  performance  in  areas  that  could  be  called  “soft”  solutions.  Evaluation  of  these 
improvements  provides  data  for  economic  analysis  that  reinforces  this  particular 
technology  as  a  positive  solution  for  current  mobility  C2  shortfalls. 

Discussion 

Overview  of  the  WIDE  and  GRS  Applications: 

WIDE  provides  individual  and  distributed  team  work-centered  user  interface  clients  that 
support  mobility  operations  with  special  emphasis  on  collaboration  across  functional  and 
supervisory  positions.  The  goal  is  to  increase  situational  awareness  during  mission 
planning  and  execution  to  allow  the  team  to  self-synchronize  and  ensure  a  highly 
coordinated  effort.  GRS  provides  functionality  that  directly  supports  WIDE  and  enables  it 
to  operate  in  its  most  effective  fashion.  Therefore,  for  the  rest  of  this  paper  the  authors  will 
often  refer  to  WIDE/GRS  as  a  combined  solution  since  the  estimated  economic  benefits 
were  calculated  based  on  both  applications  being  transitioned  together. 

WIDE/GRS  has  been  developed  in  three  spirals,  two  of  which  have  already  been  presented 
to  Air  Mobility  Command  (AMC).  GRS  will  be  included  as  part  of  the  third  Spiral. 

During  Spiral  1,  the  Air  Force  Research  Laboratory  (AFRL)  developed  and  demonstrated 
within  WIDE  mission  and  sortie  timeline  displays  using  the  emerging  Work-Centered 
Support  System  (WCSS)  cognitive-based  analysis  and  user  interface  design  approach. 
Interviews  and  observations  of  Tanker  Airlift  Control  Center  (TACC)  personnel  (Seniors, 
Duty  Officers,  Deputy  Directors  of  Operations,  Controllers,  and  Mission  Planners)  were 
used  to  provide  an  understanding  of  the  cognitive  work  involved  in  mission  planning  and 
execution.  The  cognitive  decision  making  requirements  were  analyzed  to  determine 
efficient  visual  representations  for  monitoring  and  replanning  work.  In  the  timeline 
displays,  views  of  the  missions,  sorties,  and  their  resource  constraints  were  visually 
mapped  to  a  common  timescale  that  is  updated  with  real-time  Mobility  Air  Forces  (MAF) 
mission,  location,  and  resource,  data.  Intelligent  agents  were  used  to  demonstrate 
automation  assistance  in  mission  planning  and  execution  situational  awareness.  Intelligent 
agents  used  business  rules  for  monitoring  the  data  for  factors  that  may  affect  the  mission 
plan  and  alert  the  operator  to  exceptions  through  the  visualization.  Mission  execution 
personnel  are  alerted  to  problems  in  context  and  are  able  to  see  the  factors  affecting 
mission  viability  and  offer  possible  solutions.  They  are  able  to  perform  “what-if  ’ 
simulations  to  identify  the  repercussions  of  any  changes  to  the  mission  plan. 
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Spiral  2  built  upon  the  Spiral  1  functionalities  and  used  the  WCSS  approach  to  develop  and 
demonstrate  work-centered  visualizations  to  support  dynamic  aircraft  reallocation 
decisions.  These  are  decisions  associated  with  the  in-system  select  where  the  TACC 
operators  must  select  a  resource  to  save  a  mission.  An  enhanced  Spiral  1  functionality  was 
developed  to  determine  the  best  choice  while  minimizing  the  ramifications  of  AMC’s 
overall  operations. 

Spiral  3  has  most  recently  been  demonstrated  at  AMC  and  continues  to  build  on  the  first 
two  spirals  and  use  the  WCSS  approach  to  develop  and  demonstrate  work-centered 
visualizations  to  support  dynamic  resource  reallocation  decisions  across  multiple  missions. 
Spiral  3  will  support  replanning  multiple  missions  simultaneously  and  provide 
visualizations  to  analyze  the  impact  of  the  many  changes  permeated  through  the  overall 
plan. 

Human  Factors  in  C2: 

To  more  fully  understand  why  WIDE  is  so  effective  and  presents  such  a  radical 
improvement  in  C2  capability  it  is  important  to  understand  why  human  factors 
consideration  is  so  vital.  The  following  discusses  the  underlying  cognitive  processes  for 
much  of  C2  and  suggests  that  assistance  with  these  processes  is  critical  to  effective  C2 
activity. 

Situational  awareness  (SA),  attention  span,  working  memory,  workload,  and  critical  errors 
are  all  aspects  of  human  cognition,  behavior,  and  information  processing  that  play 
important  and  complex  roles  in  C2  operations.  For  example,  each  of  the  three  levels  of 
situational  awareness:  perception,  comprehension,  and  projection  are  directly  and 
dynamically  involved  in  the  C2  operator’s  environment.  Referred  to  in  three  levels,  SA  is 
key  to  the  operator’s  successful  mission  control. 

Level  1  SA  involves  seeing  the  relevant  data.  It  simply  means  the  operator  is  looking  at 
what  is  most  appropriate.  (3) 

Level  2  SA  involved  the  ability  of  the  operator  to  understand,  or  adequately  comprehend 
what  is  being  presented.  (4) 

Level  3  involves  the  ability  to  look  forward  with  the  ability  to  understand  the  future 
situation  and  its  consequences.  (5) 

Working  multiple  global  missions  at  once,  the  C2  operator  must  be  able  to  form  an 
accurate  picture  of  what  is  happening  with  each  mission,  understand  this  information,  and 
determine  how  it  relates  to  the  mission  plans,  and  also  be  able  to  anticipate  problems,  or 
other  critical  future  events  that  could  negatively  affect  the  success  of  one  or  more  missions. 
This  is  quite  challenging  in  the  C2  operator’s  environment  which  is  replete  with  almost 
constant  distractions  that  present  challenges  to  attention  span  and  working  memory.  The 
result  of  this  high  cognitive  workload  environment  is  often  decision  errors  that  cause 
significant  problems  with  mission  progress  and  success.  Any  technology  that  will  improve 
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situational  awareness,  enhance  attention  span  and  working  memory,  decrease  actual  and 
perceived  workload,  as  well  as  reduce  critical  errors  will  dramatically  improve  mission 
success  and,  it  is  assumed,  reduce  cost  and  save  money.  The  question  is  how  does  one 
prove  it?  How  does  one  connect  the  dots  from  improvement  in  cognitive  behavior  and 
financial  benefits  realized  from  that  improvement?  One  begins  with  the  research  on 
behaviors  which  demonstrates  improvement  with  the  new  technology. 


AFRL  Evaluation  Research  on  WIDE: 

Evaluation  of  WIDE  Spiral  1  was  accomplished  by  Dr.  Emilie  Roth  and  a  team  of 
behavioral  scientists  working  with  AFRL  (7).  For  Dr.  Roth  and  her  colleagues,  a  key 
objective  of  a  work-centered  evaluation  was  to  assess  whether  the  proposed  design 
concepts,  as  embodied  in  the  prototype,  have  the  positive  effects  predicted  by  the  system 
developers  (i.e.,  to  evaluate  the  hypothesized  model  of  support).  The  specific  anticipated 
benefits  being  analyzed  included: 

a.  A  decrease  in  time  to  recognize  impacts  of  mission  changes  during  execution. 

b.  A  decreased  number  of  errors  in  replanned  missions  in  execution. 

The  evaluation  tested  these  hypothesized  benefits  by  comparing  performance  of  execution 
cell  personnel  in  the  TACC  using  the  timeline  with  their  performance  using  their  current 
legacy  system  on  comparable  scenarios.  The  key  performance  parameter  success  criteria  to 
be  established  by  the  evaluation  were: 

a.  User  acceptance  (via  post-test  questionnaire):  60%  participant  acceptance 
established  as  a  minimum  requirement,  90%  as  the  objective. 

b.  Improved  mission-related  SA:  Level  2  SA  (comprehension  of  the  current 
situation)  was  established  as  a  minimum  requirement.  Level  3  SA  (the  ability  to  project 
impact  on  future  sorties  and  missions)  was  established  as  the  objective. 

The  study  participants  consisted  of  twelve  experienced  members  of  the  execution  cell 
within  the  TACC,  which  is  AMC’s  global  C2  management  facility.  These  participants 
were  given  several  scenarios  using  both  their  legacy  system  or  WIDE.  There  were  two 
phases  in  the  test  scenarios: 

a.  An  “initial  call”  which  highlighted  aspects  of  a  mission  problem.  Subjects  were 
asked  to  evaluate  the  repercussions,  or  “broken  glass”  resulting  from,  the  problem.  The 
term  “broken  glass”  refers  to  problems  with  additional  missions  that  would  possibly  result 
from  a  solution. 

b.  Subjects  were  given  a  proposed  solution  and  asked  to  evaluate  whether  the 
recommended  solution  eliminated  the  problem(s),  or  perhaps  introduced  new  difficulties  to 
the  current  problem  set  (broken  glass). 
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Following  each  condition,  the  subjects  completed  a  National  Aeronautics  and  Space 
Administration  (NASA)  Task  Load  Index  (TLX)  workload  rating  form  and  an  SA  self- 
report  rating  form  for  that  test  condition. 

The  results  demonstrated  statistically  significant  improvements  in  response  time  between 
the  existing  legacy  system  and  the  WIDE  timeline.  Errors  were  also  found  to  be  reduced 
with  WIDE  vice  the  legacy  system.  Mean  SA  ratings  were  higher  for  WIDE  than  the 
legacy  system  and  included  some  Level  3  SA.  Additionally,  WIDE  exceeded  the 
established  key  performance  parameters  objectives.  90%  of  the  users  found  WIDE 
acceptable  and  agreed  it  improved  Level  3  SA.  Finally,  perceived  workload  was  lower  at  a 
statistically  significant  level  with  WIDE  as  opposed  to  the  legacy  system.  The  graph  below 
highlights  the  results  of  the  improvements  in  SA. 


Rating 


■  i™ 


Mission  Temporal  Impact  Impact  Impact  Plan 

Plan  Constraints  On  On  On  Changes 

Current  Future  Future  Needed 

Sortie  Sorties  Missions 


■  Timeline 

■  Legacy  System 


Elements  of  Situational  Awareness 


For  the  evaluation  of  WIDE  Spiral  2,  Dr.  Roth  and  her  team  focused  on  a  more  complex 
cognitive  problem:  reallocation  of  aircraft  to  support  real-time  dynamic  replanning  of 
missions.  Projected  benefits  for  Spiral  2  included: 

a.  Increased  SA  for  viable  courses  of  action  (COAs)  for  short  notice  missions. 

b.  Increased  effective  resource  allocation  decisions. 

c.  Minimize  the  negative  impacts  of  reallocation  decisions  (“broken  glass”). 

Similar  protocols  were  followed  for  the  Spiral  2  research  including  subjects  using  both  the 
legacy  system  and  the  WIDE  Resource  Candidate  (ReCaD)  displays  (8).  The  results 
demonstrated  100%  of  the  subjects  rated  ReCaD  as  acceptable  (objective  was  60%  as 
minimum  and  90%  as  objective).  Improvements  in  situational  awareness  and  decision 
quality  were  documented  as  was  a  positive  impact  on  a  variety  of  C2  operator  positions 
within  the  TACC. 
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Importance  of  Measuring  “Soft"  Benefits: 


While  one  may  intuit  that  improvement  in  these  cognitive  processes  associated  with  C2 
operations  will  produce  benefits,  actual  measurement  demonstrates  the  validity  of  this 
assumption.  However,  in  today’s  fiscally  constrained  environment  one  must  go  beyond 
demonstration  of  these  benefits  to  adequately  justify  acquisition  consideration  and 
investment. 

Connecting  the  Operational  and  Fiscal  Dots: 

In  order  to  prove  the  financial  benefit  of  acquiring  a  technology  that  demonstrates  the 
above-mentioned  human  cognitive  and  mission-related  improvements,  there  are  challenges 
that  need  to  be  overcome.  While  there  are  identified,  plausible  benefits  from  WIDE 
timeline  functionality,  they  are  improvements  in  areas  such  as  situational  awareness  and 
reduced  critical  errors.  Even  though  these  improvements  are  definitely  important  and 
speak  directly  to  C2  needs  that  were  identified  in  the  MAF  C2  FNA,  they  are  difficult  to 
quantify  from  a  cost/beneflt  perspective. 

In  many  other  areas,  benefits  are  more  easily  calculated  because  they  result  from  what 
might  be  called  “hard  improvements.”  For  example,  a  cost/benefit  calculation  might  be 
done  on  a  new  jet  engine  to  determine  the  financial  benefit  received  from  improved  fuel 
economy.  The  engine  can  be  tested  and  the  improvement  in  terms  of  reduced  fuel  use 
documented  in  hard  numbers.  Thus,  a  cost/benefit  calculation  is  rather  simple  because  the 
demonstrated  benefit  is  easily  seen  via  the  test  of  equipment  and  is  presented  in  numbers 
resulting  from  the  engine  test.  In  the  case  of  C2  benefits,  the  improvements  are  not  always 
immediately  evident.  How,  for  example,  does  improvement  in  situational  awareness 
translate  into  dollars  saved?  This  was  the  challenge  faced  by  the  investigators. 

In  order  to  perform  a  cost/benefit  calculation  on  WIDE/GRS  functionality,  “soft” 
improvements  from  this  functionality  must  be  identified.  This  identification  was  realized 
from  analysis  of  the  capabilities  needs,  i.e.,  what  the  MAF  should  be  doing  better  and  its 
importance,  and  how  documented  improvements  to  such  things  as  situational  awareness 
related  to  the  capabilities/needs.  The  source  for  these  improvements  was  the  above- 
mentioned  study  by  Dr.  Roth.  Each  of  the  areas  which  WIDE/GRS  functionality  was 
found  to  improve  could  be  translated  into  some  dollar  savings.  For  example,  the  ability  for 
mission  managers  to  retask  aircraft  more  quickly  can  result  in  more  efficient  tasking  and 
fewer  delays  according  to  subject  matter  experts  (SMEs)  who  were  interviewed  for  the  cost 
benefit  evaluation.  This  can  be  translated  into  actual  dollars  per  flying  hour  saved 
assuming  eliminating  or  mediating  one  delay  can  result  in  the  savings  of  one  flying  hour 
(as  projected  by  SMEs).  Using  this  type  of  premise,  soft  improvements  can  be  translated 
into  dollar  cost  savings  or  cost  avoidance. 

Unfortunately,  no  objective  research  evidence  is  known  to  be  available  which  will  validate 
these  assumptions  beyond  SME  inputs,  it  argues  for  additional  research  to  be  done 
concerning  how  efficiencies  in  planning  and  execution  can  positively  affect  airlift  mission 
success. 
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The  benefits  and  costs  of  the  proposed  WIDE/GRS  solution  were  systematically  identified 
and  roughly  quantified  in  dollar  terms.  An  organizing  structure  was  prepared  to  display 
and  compare  the  dollar  benefits  and  costs  throughout  an  eight  year  period.  This  structure 
allows  for  the  calculation  of  Benefit/Cost  (B/C)  ratios  and  Return  on  Investment  (ROI) 
values. 

Case  Study:  A  Tailored  Cost-Benefit  Approach 

Purpose  of  the  Cost-Benefit  Calculation: 

Within  the  CBP  C2  community,  WIDE/GRS  functionality  has  been  identified  as  a 
promising  candidate  solution  for  improving  mission  planning  and  management  activities. 
Before  any  cost-benefit  calculations  were  done,  consensus  impressions  were  that  WIDE 
functionality  would  offer  significantly  better  situational  awareness,  and  thus  lead  to  better 
performance  in  areas  such  as  decision-making,  coordination,  and  synchronization. 

The  main  purpose  of  the  cost-benefit  calculation  was  to  evaluate  these  consensus 
impressions,  by  specifying  and  quantifying  the  expected  impacts  of  WIDE/GRS,  in  dollar 
terms.  Does  WIDE/GRS  make  financial  sense,  consistent  with  expected  improvements  in 
“situational  awareness”?  At  each  round  of  quick  and  rough  calculations,  WIDE/GRS 
showed  impressive  positive  dollar  impacts.  The  consensus  impressions  were  reinforced,  in 
financial  terms. 

Conventional  Cost-Benefit  Evaluations: 

Even  though  WIDE/GRS  poses  various  “soft”  benefits,  conventional  cost-benefit  practices 
provide  a  useful  structure  for  systematically  presenting  costs  and  benefits.  Once  specified 
and  captured,  the  calculations  are  mostly  straight-forward.  At  a  generic  level,  the  structure 
includes  four  blocs,  to  display  information  about  the: 

Period  of  Performance 

Costs 

Benefits 

Summary  Calculations. 

Of  course,  nuances  such  as  discounting  and  present  value,  along  with  the  selection  of 
inflation  factors,  play  important  roles  in  formal  cost-benefit  analyses  (also  known  as 
economic  analysis).  This  paper,  however,  deals  with  the  “first  cut”  version  of  the  study. 

Its  emphasis  is  on  how  we  managed  to  specify  the  ethereal  benefits  more  concretely,  which 
allowed  us  to  quantify  impacts  in  conventional  dollar  terms. 

The  use  of  the  conventional  cost-benefit  structure  also  provides  a  template  for  other  users 
who  face  similar  problems  in  advocating  proposed  solutions  which  are  characterized  by 
“soft”  benefits. 
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Unique,  Tailored  Features  of  this  Calculation: 


At  least  two  features  make  this  study  somewhat  unusual.  First,  a  brainstormed  list  was  the 
essential  step  to  eventually  specify  three  “soft”  benefits  (to  Mission  Managers,  to  Tails 
during  Operations,  and  to  Aerial  Port  Activities).  Second,  results  from  experiments  (using 
an  early  version  of  WIDE/GRS,  with  actual  mission  planners  and  managers)  provided  data 
to  help  quantify  WIDE/GRS’s  impacts  on  behavior  and  performance. 

Mission  Managers  —  Specifying  the  benefits  enough  to  quantify  their  impacts  was  the  first 
major  issue.  The  technique  of  brainstorming  an  extensive  listing  of  potentially  positive 
impacts  was  used.  Close  scrutiny  of  the  list  indicated  that  most  of  the  benefits  were 
associated  with  a  particular  set  of  activities. . .  by  mission  managers!  It  became  apparent 
that  WIDE/GRS  could  provide  mission  managers  with  solutions  much  more  quickly  than 
the  mission  managers  could  generate  manually  (as  in  the  status  quo).  This  specified  the 
type  of  benefit  more  concretely,  and  allowed  for  gathering  estimates  for  the  amount  of  time 
that  WIDE/GRS  would  save  for  each  activity.  Along  with  data  and  assumptions  regarding 
frequencies  of  occurrence  of  the  activities,  and  estimates  of  the  time  costs  of  mission 
managers,  “soft”  benefits  were  credibly  quantified  into  dollar  terms.  These  savings  of 
hours  and  minutes  on  individual  activities  grow  to  large  dollar  impacts  when  multiplied  by 
the  approximately  60,000  missions  that  AMC  has  flown  annually  in  recent  years. 

Tails,  during  Operations  —  Discussions  around  the  brainstormed  list  led  easily  to  the 
possibility  that  improved  planning  and  mission  management  through  “soft”  WIDE/GRS 
benefits  could  prevent  at  least  some  delays!  This  resultant  benefit  was  specified  as  “Tails, 
during  Missions.”  Tails  and  Flying  Hours  could  be  saved.  With  data  and  assumptions 
about  frequencies  of  delays,  and  costs  of  Flying  Hours,  large  dollar  savings  again  result 
from  the  many  missions  flown  each  year. 

Aerial  Port  Activities  —  Similarly,  personnel  and  expensive  Materiel  Handling  Equipment 
(MHEs)  at  Aerial  Ports  can  also  be  managed  more  effectively.(9)  Idle  time  waiting  to 
unload/load  unexpectedly  late  aircraft  is  reduced  by  WIDE/GRS’s  improvements  to  the 
retasking  activity  (thus,  fewer  late  arrivals).  With  Halverson  MHEs  ($640,000  each)  and 
Tunners  ($1.7  million),  and  multiple  personnel  assigned  per  load  event,  idle  time  is 
expensive  (and  quantifiable).  The  authors  suggest  that  even  more  dollar  savings  could  be 
calculated. 

Sources  —  After  “soft”  benefits  are  identified  and  specified  more  concretely  (as  above),  a 
following  task  for  the  analyst  (and  proponents)  is  to  get  answers  to  the  question,  “So 
what?”  In  terms  of  time  or  dollars,  how  much  impact  does  a  proposed  solution  like 
WIDE/GRS  have?  A  numerical  value  is  called  for,  preferably  in  terms  of  time  and/or 
dollars  (or  $/Hr).  Credible  numerical  values  of  a  proposed  solution’s  positive  impacts  can 
provide  powerful  rationale  for  acquiring  the  solution.  Conventional  sources  include 
historical  data,  analogy,  inputs  from  Subject  Matter  Experts  (SMEs),  and  placeholder 
guesstimates.  Data  values  from  any  of  these  usual  source  types  can  be  valuable  and 
persuasive. 
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WIDE/GRS  developers  (AFRL)  delivered  an  extraordinary  source  of  data  values.  They 
conducted  surveys  of  performances  by  mission  managers. .  .with  a  prototype  WIDE  vs. 
without  WIDE.  The  results  demonstrated  impressive  improvements  with  WIDE.  Response 
times  were  quicker,  critical  errors  were  fewer,  situational  awareness  was  greater,  and 
perceived  workloads  lower.  These  results  were  consistent  with  data  values  that  were 
gleaned  from  historical  data  and  from  SMEs.  These  fortuitous  experimental  findings 
reinforce  the  credibility  of  SME  input  values,  and  thus  confidence  in  the  calculations. 

Calculation  Results: 

Actual  values  of  inputs  and  results  are  not  included  in  this  paper.  However,  the  dummy 
values  do  represent  the  relationships  and  conclusions  of  the  calculations. 

Costs  to  transition/implement  WIDE/GRS  were  expectably  frontloaded,  and  relatively  low 
because  it  is  an  application  to  be  incorporated  into  an  existing  system.  A  risk  area,  as 
usual,  lies  in  the  integration  of  the  necessary  data  exchanges.  Development  and 
procurement  costs  are  borne  by  AFRL,  and  thus  not  included  in  the  Cost-Benefit 
calculations. 

The  three  types  of  benefits  yield  positive  time  and  dollar  impacts  from  the  first  year,  and 
depend  on  the  number  of  sorties/missions  flown.  Because  of  the  numerous  missions,  the 
dollar  value  of  benefits  overwhelms  costs  almost  immediately,  and  continues  to  dominate 
throughout  the  period  of  performance. 

The  consistently  favorable  benefit-to-cost  ratios  presented  (with  representative  dummy 
values)  begin  at  more  than  1  in  the  first  year,  and  soon  increase  to  more  than  30.  The 
Return  on  Investment  values  (net  accumulated  benefits)  also  surge  in  year  3,  and  likewise 
climb  to  over  30.  WIDE/GRS  clearly  appears  to  make  solid  financial  sense. 

Afterwords 

More  Benefits  are  Out  There  —  We  noticed  that  there  were  additional  benefits  to  be  had 
beyond  those  in  the  three  categories  (to  Mission  Managers,  to  Tails  during  Operations,  and 
to  Aerial  Port  Activities)  that  we  initially  specified  and  quantified.  Further  reductions  in 
errors,  delays  and  in  time  requirements  might  be  achieved  in  several  more  areas.  Crew 
Management  (and  Availability),  for  example,  would  benefit  from  fewer  delays  in 
operations.  Even  customers  might  benefit,  by  the  lessening  of  Customer  Wait  Times 
(again,  through  WIDE-enabled  fewer  delays).  The  dollar  value  of  Fuel  Savings  (fewer 
Flying  Hours,  from  fewer  diverts  or  delays)  can  be  quantified  and  highlighted.  Each  of 
these  additional  types  of  benefits  is  quantifiable,  and  is  a  candidate  to  be  included  in  future 
versions  of  cost-benefits  for  this  and  other  C2  solutions. 

A  particularly  expected  valuable  benefit  from  WIDE/GRS  lies  in  its  capability  to 
immediately  generate  and  present  “broken  glass”  (/'.<?.,  the  multiple  delays  or  cancellations 
of  other  missions  which  result  from  an  initial  retasking).  Thus,  Mission  Managers  will  be 
able  to  see  and  evaluate  the  secondary  and  tertiary  impacts  of  alternative  retasking  choices. 
Future  benefit-cost  calculations  will  address  the  enormous  dollar  implications  of  “broken 


10 


glass.”  All  C2  operators  are  now  able  to  handle  increased  numbers  of  missions  that  are  of 
more  significant  complexity  than  they  were  able  to  effectively  manage  in  the  past. 


Benefits  Already  Addressed: 

Mission  Managers 
Tails,  during  Operations 
Aerial  Port  Activities 

Candidate  Benefits  for  Future  Inclusion: 

Crew  Management  &  Availability 
“Broken  Glass” 

Customer  Wait  Time 
Fuel  Savings 

Other  Benefits  Possible  with  Additional  Research  -  This  paper  has  thus  far  mainly 
discussed  benefits  that  are  reactionary  in  nature.  That  is,  they  are  obvious  results  of 
changes  in  user  behavior  as  a  result  of  the  introduction  of  a  technology  that  supports  their 
cognitive  work.  Given  the  improvements  that  have  been  documented  via  AFRL’s  research, 
it  would  lead  one  to  consider  how  the  C2  process  could  be  affected  by  the  introduction  of  a 
technology  that  improves  the  operator’s  ability  to  handle  work  of  more  significant 
cognitive  complexity. 

C2  capabilities  and  processes  have  been  suggested  in  MAJCOM  and  Air  Force  Master 
Capabilities  Lists  based  on  current  technology  and  a  human  -  computer  interface  that  has 
not  kept  pace  with  research  into  technologies  that  enhance  the  cognitive  aspects  of  work.  It 
is  worth  consideration  for  research  to  be  done  that  would  investigate  how  C2  processes 
could  be  improved  based  on  the  technological  improvements  in  the  human  -  computer 
interface.  For  example,  can  planning  tasks  be  expanded  based  on  a  system’s  ability  to 
support  the  planner?  Is  it  possible  to  erase  the  planning/execution  gap  and  make  the 
actions  seamless  as  they  transfer  one  to  another?  Can  strategic  level  planning  be  expanded 
to  include  collaboration  between  actors  in  real  time  that  would  result  in  a  strategic  level 
planning  document  that  could  immediately  be  morphed  into  operational  and  tactical  plans? 
These  questions  and  other  should  be  considered  for  additional  research  opportunities. 

This  Approach  is  Do-able  by  Others  —  Future  Cost-Benefits  exercises  for  C2  solutions  may 
be  carried  out  by  any  proponent.  A  rough,  quick,  initial  version  may  use  the  standard 
template,  and  the  “list”  approach  to  specifying  “soft”  benefits.  Finally,  quantification  in 
dollar  terms  is  accomplished  through  data  (especially  time  and  $/Hr),  SME  inputs,  and 
experimental  results  (if  available).  Please  let  us  know  what  you  find. 
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ability  of  operators  managing  mission  execution  to  perform  dynamic  replanning  in 
support  of  short  notice  requirements.  The  planner  or  person  in  execution  will  use  new 
search  functions  and  displays  that  include  a  tabular  presentation  of  possible  aircraft 
resources  (resource  candidates).  It  includes  a  coordinated  map  showing  the  geographic 
location  of  possible  aircraft  tails  and  an  expanded  Multi-Mission  Detail  Timeline 
visualization  that  enables  individuals  to  examine  mission  details  and  explore  the 
ramifications  of  redirecting  resources  to  support  the  short  notice  requirement. 

9.  Materiel  handling  equipment  (MHE)  refers  to  the  family  of  large  wheeled  vehicles 
which  are  used  for  the  movement  and  storage  of  materiel  within  a  facility  or  a  site.  MHE 
range  in  size  and  capacity  from  a  wheeled  pry-bar  used  to  lift  one  side  of  a  100  lb.  box  all 
the  way  up  to  the  25,000  pound  capacity  (Halvorsen)  and  60,000  pound  capacity 
(Tunner)  cargo  loaders.  These  loaders  are  used  to  transport  cargo  between  aircraft  and 
storage/staging  areas  and  are  the  primary  method  of  loading  and  unloading  cargo  aboard 
a  variety  of  AMC  aircraft  and  AMC-contracted  commercial  aircraft.  Both  loaders  are 
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capable  of  transporting  a  variety  of  types  of  cargo  including  palletized  cargo,  shipping 
containers,  airdrop  platforms,  and  rolling  stock  (e.g.,  vehicles,  carts,  etc.).  Both  loaders 
are  also  designed  to  be  transported  by  air  on  C-5  and  C-17  aircraft  and  can  be  moved  be 
surface  on  a  truck,  rail  or  ocean  vessel.  (Tom  Gross,  SME  at  HQ  AMC/A3R;  14  March 
2008.) 
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GLOSSARY 


AFRL 

Air  Force  Research  Laboratory 

AMC 

Air  Mobility  Command 

B/C 

Benefit/Cost 

C2 

Command  and  Control 

CBP 

Capabilities  Based  Planning 

COA 

Course  of  Action 

DOTMLPF 

Doctrine,  Organization,  Training,  Leadership,  Policy, 
and  Facilities 

FAA 

Functional  Area  Analysis 

FNA 

Functional  Needs  Analysis 

FSA 

Functional  Solutions  Analysis 

MAF 

Mobility  Air  Forces 

MHE 

Materiel  Handling  Equipment 

NASA 

National  Aeronautics  and  Space  Administration 

ReCad 

Resource  Candidate  Displays 

ROI 

Return  on  Investment 

SA 

Situational  Awareness 

SME 

Subject  Matter  Expert 

TACC 

Tanker  Airlift  Control  Center 

TLX 

Task  Load  Index 

WCSS 

Work-Centered  Support  System 

WIDE 

Work-Centered  Interface  Distributed  Environment 
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Work  Centered  Interfaced  Distributed 

Environment 

(WIDE) 


ion  Planning 


Mission  Plan 


Real-Tirrn 


Mission  Data 


Work-Centered  Upgrades  to  the  Tanker  Airlift  Control  Center 

m . .  ■ 


Cognitive  Work 
Support  Suite 


Exceptions 
Reported  in 
Work  Context 


Visualizations  Aid 


jcjL 


Monitors*MissiorrResources 


THE  TECHNOLOGY: 

Work-Centered  Support  Systems 

•  Supports  Cognitive  Aspects  of  Work 

•  Cognitive  Based  Visualization  & 
Automation 

•  Human-Centered  Upgrades  to  C2  Systems 
AFRL/RI  Multi-Mission  Optimization 


WHAT  IT  IS: 

Advanced  Human-Computer  Interfaces  (HCI)  to 
Plan,  Monitor,  and  Synchronize  Global  Airlift 
Missions. 
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Introduction 
to 

Cost-Benefit  Calculation 

■  A  Period  of  Performance 

■  A  Template  Structure 

■  A  List  of  Benefits  and  a  List  of  Costs 

■  Estimated  or  Actual  Data  Values 

■  Calculation  Display 

■  Essential  Lessons 
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Template  Structure 


PERIOD  OF  PERFORMANCE 

COSTS 

o 

BENEFITS 

. 

,  7  ctt'' 

AO* 

Four  Blocks 

Ac*'° 

1 

_ +  _ 

SUMMARIES 

9 
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Template  Structure 


PERIOD  OF  PERFORMANCE 

2008  2009  2010  2011  2012  2013  2014  2015 

COSTS 

Development 

Integration 

Sustainment 

BENEFITS 

Mission  Managers 

Tails  During  Operations 

Aerial  Port  Activities 

SUMMARIES 

Cost-Benefit  Ratios 
Return  on  Investment 
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2  LISTS  -  Initial  &  Refined 


Brainstormed  List  of  Benefits 

>  Improved  Situational  Awareness 

>  Situational  Awareness  of  Mission  Details 

>  Improved  Situational  Awareness  at  Focal  Location 

>  Improved  Situational  Awareness  of  Problems 
Resulting  from  Mission  Changes 

>  Ability  to  Identify  Aircraft  for  Tasking 

>  Ability  to  Develop  a  Feasible  Plan 

>  Reduces  Decision  Making  Time 

>  Improves  Decision  Making  by  Allowing  Fewer 
Negative  Consequences 

>  Reduces  the  Possibility  of  Errors 

>  Improves  Efficiency  of  Resource  Utilization 

>  Better  Decisions  for  Short  Notice  Missions 

>  Reduced  Operator  Workload 


Improved  Granularity 


Consolidated/Refined  List 


Initial  Thoughts  of  SMEs  and  Others 


Lengthy  First  Take 


Refined  List  of  Benefits 

>  MISSION  MANAGERS 

>  Dynamic  Retasking 

>  Situational  Awareness 

>  Fewer  Critical  Errors 

>  Improved  Decision  Making 

>  TAILS  DURING  OPERATIONS 

>  Fewer  Delayed  Missions 

>  AERIAL  PORT  ACTIVITIES 

>  Asset  and  Resource  Management 
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The  Values 


Estimated  or  Actual  Data  Values 


Expressed  In: 


Sources: 

S  History 
S  SMEs 

S  Placeholder  WAGs 
S  Experimental  Findings 
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Calculation  Display 

(Not  Actual  Values) 


COSTS 

2008 

2009 

2010 

2011 

2012 

2013 

2014 

2015 

Dev 

$ 

$ 

$ 

$ 

$ 

$ 

$ 

Integration 

$  20 

$  10 

Sustainment 

$  10 

$  15 

$  35 

$15 

$35 

$15 

$15 

$15 

ANNUAL  $  COSTS 

$  30 

$  25 

$  35 

$  15 

$  35 

$  15 

$  15 

$  15 

BENEFITS 

2008 

2009 

2010 

2011 

2012 

2013 

2014 

2015 

Mission  Manager 

5 

10 

150 

150 

150 

150 

150 

150 

Tails  during  Opns 

50 

$  100 

$1,000 

$1,000 

$  1,000 

$1,000 

$1,000 

$1,000 

Aerial  Port  Opns 

2 

$  5 

$  50 

$  50 

$  50 

$  50 

$  50 

$  50 

ANNUAL  $  BENEFITS 

$  57 

$  115 

$1,200 

$1,200 

$  1,200 

$1,200 

$1,200 

$1,200 

SUMMARIES 

Benefit  /  Cost  Ratio 

1.9 

4.6 

34.3 

80.0 

34.3 

80.0 

80.0 

80.0 

Return  on  Investment 

1.9 

3.1 

15.2 

24.5 

26.9 

32.1 

36.3 

39.8 
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Quantifying  “Soft”  Benefits 

(Nailing  Jell-0  to  the  Wall) 


How  Does  Improved 
Cognitive  Performance 


Improved 
Job  Performance 
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From  “Soft” 
to 

Measurable 


COGNITION 

>  Improved  Situational  Awareness 

>  Reduced  Mental  Workload 


PERFORMANCE 

>  Faster  Answers 

>  Fewer  Errors 


V 


IMPACTS  ON: 

>  Mission  Managers 

>  Tails  during  Opns 

>  Aerial  Port  Activities.... 


Quicker  Re-Tasking  of  Missions 
Fewer  Delays 
ess  Idle  Time" 
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...from  Measurable  ...to  Impacts 
(It’s  Mostly  Arithmetic) 


IMPACTS  ON: 

Variables  (mainly  TIME!) 
Needed  to  Calculate 

“Soft”  Benefits 
QUANTIFIED  into  $$$ 

MISSION  MANAGERS: 

Quicker  Retaskings 

$  per  Mission  Manager  Hour 
#  Retaskings  per  Year 

Hrs  to  Deliver  a  Retaskina  Order  (w/o  WIDE) 

Hrs  to  Deliver  a  Retasking  Order  (w/WIDE) 

Annual  $$  Value  of  FEWER  MSN 

MANAGER  RETASKING  HOURS 

....w/WIDE 

TAILS  during  OPERATIONS: 

Fewer  Delays 

$/Flying  Hour 

#  MSNs  per  Year 

%  MSNs  Delayed/Yr  (w/o  WIDE) 

#  MSNs  Delayed/Yr  (w/oWIDE) 

Ava.  Hrs  per  Delayed  MSNs/Yr  (w/o  WIDE) 

Annual  $$  Value  of 

AVOIDED  DELAYS 
(i.e.,  FLYING  HOURS) 

....w/WIDE 

%  MSNs  Delayer/Yr  (w/WIDE) 

#  MSNs  Delayed/Yr  (w/WIDE) 

Avg.  Hrs  per  Delayed  MSNs/Yr  (w/WIDE) 

AERIAL  PORT  OPERATIONS: 

Less  Idle  Time 

MHr  Cost 

#  People  per  Crew 

MsnHandlingEquipment  Hourly  Cost 

#  MHEs  per  MSN  Load 

MHrs  Idle  Time  per  Msn  (w/o  WIDE) 

MSNs  per  Year 

Annual  $$  Value  of 

IDLE  TIME  AVOIDED 

....w/WIDE 

%  Idle  Time  Reduced  (w/WIDE)  per  MSN 
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More  Types  of  “Soft” 
Benefits  to  Exploit 
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Essential  Lessons 


Collaborate!  Brainstorm! 


Use  a  Template  -  with  Feeders 


For  Data  Values,  Make  Generous  Use  Of: 

>  SME  Inputs 

>  Experimental  Results 

>  Placeholder  WAGs 


Collaborate! 


It’s  Flexible!  It’s  Tailorable! 

>  Useful  for  Other  Proponents 
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Bottom  Lines 


To  Win  the  Budget  Fight  You  Need  a  One-Two  Punch!! 
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Questions? 


Questions? 

No,  You  Don’t  Have 
Any  Questions! 
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